Image capturing device, image capturing method, program, and integrated circuit

ABSTRACT

An image capturing device includes: a pre-capturing module which captures an imaging target at each of different focus positions in a predetermined focus range and outputs, as a capturing result, a plurality of pre-captured images lower in resolution than the output image; an object sharpness evaluating module which computes a sharpness level of each of the pre-captured images captured by the pre-capturing module; a focus varying range determining module which determines a focus position varying range within the predetermined focus range based on sharpness levels computed by the object sharpness evaluating module, such that a signal-to-noise ratio of the output image is greater than or equal to a predetermined threshold; and an image capturing module which captures the imaging target while varying the focus position according to the focus position varying range determined by the focus varying range determining module, and outputs the output image as a capturing result.

BACKGROUND OF INVENTION Technical Field

The present invention relates to image data processing. Morespecifically, the present invention relates to a method and a device forgenerating an image having an extended depth of field from at least oneimage captured while focus position is varied.

BACKGROUND ART

An image with a large depth of field is a relatively deep 3-dimensional(3D) image in which both relatively near and relatively far objectsappear acceptably sharp (in-focus). A large depth of field is preferredin many situations. For example, in 3D applications of an image of aperson standing in front of beautiful background scenery, it ispreferable to see the image of both the person and the beautifulbackground scenery sharply in 3D (i.e. stereoscopic 3D). This allows theviewer to sharply see the 3D image both in the case of seeing the personat the front and in the case of seeing the beautiful background scenery.

Conventionally, a simple method for increasing the depth of field isachieved by increasing F-number of aperture (reducing the aperturediameter) of an image capturing device. However, it reduces the amountof incoming light to be captured. This causes the captured image inparticular to appear noisy, and slower shutter speed may cause handshaking and object blur in some cases.

Methods for increasing the depth of field are disclosed in PatentLiteratures 1 and 2 below. Patent Literature 1 (the specification ofU.S. Pat. No. 6,201,899, issued on Mar. 13, 2001) discloses a method andan apparatus for extended depth of field imaging. This method usesmultiple source images, captured at different focus positions, tocombine into a single image. Referring to FIG. 2, this method receivesmultiple source images (I₁ to I_(M)) captured at different focuspositions. High pass filter is applied to each source image to obtainrelatively high frequency components. Then, the method computes energylevels of the high frequency components for each source image andselects multiple sub-regions having greatest energy levels. An extendeddepth of field image is finally constructed by combining sub-regionscorresponding to the source images having greatest energy levels of highfrequency components.

The method, however, requires that the complete multiple source imagesbe received before the image constructing can start. Thus, to obtain anextended depth of field image using this method, the above requirementoften results in a time lag of at least equal to a time required forcapturing multiple source images. Such a time lag is unacceptable insituations where real-time operation is desired for obtaining anextended depth of field image. Moreover, the method generally requires ahuge memory for storing the source images and the filtered images,especially in the case of using high resolution source images.

Patent Literature 2 (the specification of U.S. Pat. No. 7,711,259,issued on May 4, 2010) discloses a method and an apparatus forincreasing the depth of field of an image. Referring to FIG. 3, thismethod captures multiple images at different focus positions andcombines the captured images to form a final image. Then the methodsharpens the final image to construct an output image having an improveddepth of field. The method, however, is also deficient in that multipleimages are used. That is to say, a time lag may occur. This method isthus not suitable for real-time application.

In another embodiment of Patent Literature 2, another method is alsoproposed. This method captures only one image while varying the focusposition during image capturing, to form a final image. Then the methodsharpens the final image to construct an image having an improved depthof field. The range of varying the focus position can be set manually bythe user or predefined in the camera. This method is therefore not fullyautomatic and it is difficult for an average user to adjust the range.Moreover, this method is not suitable when the scene to be captured isunknown or unpredictable.

Furthermore, Non-Patent Literature 1 (“Flexible Depth of FieldPhotography”, S. Kuthirummal, H. Nagahara, C. Zhou, and S. K. Nayar,IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 99,March, 2010) discloses a method to capture one image while varying thefocus position during image capturing. These known methods predeterminethe range of varying the focus position. Referring to FIG. 1, some ofthe known methods always start the image capturing (open the shutter) byfocusing at Z₁ and move the focus position to Z₃ before stopping theimage capturing (close the shutter). This method is suitable when thescene is known beforehand or the image capturing environment is known,such as in the case of image capturing using an electronic microscope,where the distance from the camera to the subject is known. However,these methods are not suitable when the type of scene to be captured isunknown or unpredictable.

CITATION LIST Patent Literature

-   [PTL 1] Specification of U.S. Pat. No. 6,201,899-   [PTL 2] Specification of U.S. Pat. No. 7,711,259

Non Patent Literature

-   [NPL 1] “Flexible Depth of Field Photography”, S. Kuthirummal, H.    Nagahara, C. Zhou, and S. K. Nayar, IEEE Transactions on Pattern    Analysis and Machine Intelligence, Vol. 99, March, 2010-   [NPL 2] “Digital Image Processing”, R. C. Gonzalez & R. E. Woods,    Addison-Wesley Publishing Company, Inc., 1992-   [NPL 3] “Acceleration of Iterative Image Restoration    Algorithms”, D. S. C. Biggs, Applied Optics, Vol. 36, No. 8, pp.    1766-1775, 1997

SUMMARY OF INVENTION

It is an object of the present invention to provide a method and adevice for extending the depth of field of an image of a scene. Inparticular, it is an object of the present invention to provide a methodand a device that are efficient, use less memory, have no significanttime lag, capable of real-time operation, and capable of automaticallyproducing a sharp image for various scene types even when the scene tobe captured is unknown and unpredictable.

To achieve the above objects, an information processing device accordingto an aspect of the present invention is an image capturing device whichcaptures an imaging target while varying a focus position and outputs atleast one output image as a capturing result, the image capturing deviceincluding: a pre-capturing unit configured to capture the imaging targetat each of different focus positions in a predetermined focus range andoutput, as a capturing result, a plurality of pre-captured images lowerin resolution than the output image; a sharpness level computing unitconfigured to compute a sharpness level of each of the pre-capturedimages captured by the pre-capturing unit; a focus varying rangedetermining unit configured to determine a focus position varying rangewithin the predetermined focus range based on sharpness levels computedby the sharpness level computing unit, such that a signal-to-noise ratioof the output image is greater than or equal to a predeterminedthreshold; and an image capturing unit configured to capture the imagingtarget while varying the focus position according to the focus positionvarying range determined by the focus varying range determining unit,and output the at least one output image as a capturing result.

Here, it is preferable that the image capturing device further includean extended depth of field image deriving unit configured to derive anextended depth of field image having an extended depth of field, usingthe at least one output image outputted by the image capturing unit andat least one predetermined point spread function.

With this, before the imaging target is actually captured to generate anoutput image, pre-captured images are obtained by capturing the imagingtarget plural times, each time with a different focus position, and thesharpness levels of the pre-captured images are evaluated. Thus, sincethe sharpness levels of the pre-captured images captured at the pluralfocus positions are evaluated in advance, the position, relative to theimage capturing device, of an object in the scene which is the imagingtarget can be known. With this, the focus varying range determining unitcan automatically determine a focus position varying range such that therange includes the focus position of each object at which the object canbe captured sharply in the scene to be captured, for example.

As described, since, prior to actual image capturing, a focus positionvarying range is automatically determined according to the scene to becaptured, it is unnecessary for the user to manually adjust the focusposition varying range to suit the scene. This allows the user to easilyoperate the image capturing device with less confusion about how tooperate the device. Therefore, a high-quality, sharp, and clear imagewith an extended depth of field can be obtained for various scene typeseven when the position, relative to the image capturing device, of anobject in the scene to be captured is unknown.

Moreover, the pre-captured images obtained to determine a focus positionvarying range are lower in resolution than an output image, and thusfewer high resolution images need to be used and stored. Therefore,fewer memories are required than the conventional technique disclosed inPatent Literature 1. Moreover, the present invention has shorter timelags and is capable of real-time applications.

Here, it is preferable that the focus varying range determining unit beconfigured to: obtain, for each pre-captured image, a sharpness level ofeach of regions obtained by dividing each pre-captured image; comparethe sharpness levels between the pre-captured images for each region;obtain, for each region, a focus position corresponding to apre-captured image having a maximum sharpness level among the sharpnesslevels of the regions obtained for each pre-captured image; anddetermine the focus position varying range such that the obtained focuspositions are included.

With this, the focus varying range determining unit obtains, for eachpre-captured image, a sharpness level of each of regions obtained bydividing each pre-captured image, and obtains, for each region, a focusposition corresponding to a pre-captured image having a maximumsharpness level among the sharpness levels of the regions obtained foreach pre-captured image. Then, the focus varying range determining unitdetermines the focus position varying range such that the obtained focuspositions are included.

Thus, for each region, a focus position corresponding to thepre-captured image having the maximum sharpness level is obtained, and afocus position varying range is determined based on the focus positionsobtained. With this, in the case where an object is present in a regionobtained by dividing each pre-captured image, the focus positioncorresponding to that object can be obtained. In other words, because afocus position varying range is determined such that the obtained focuspositions are included, it is possible to obtain an image in which eachobject included in the scene to be captured appears sharply.

Here, it is also possible that the focus varying range determining unitis configured to: obtain, for each pre-captured image, a sharpness levelof each of regions obtained by dividing each pre-captured image; obtaina plurality of focus positions at which pre-captured images having,among the obtained sharpness levels of the pre-captured images and theregions, sharpness levels higher than a predetermined threshold havebeen captured; and determine the focus position varying range such thatthe obtained focus positions are included.

With this, the focus varying range determining unit obtains, for eachpre-captured image, a sharpness level of each of regions obtained bydividing each pre-captured image, and obtains a plurality of focuspositions at which pre-captured images having, among the sharpnesslevels obtained for each pre-captured image and each region, sharpnesslevels higher than a predetermined threshold have been captured. Then,the focus varying range determining unit determines a focus positionvarying range such that the obtained focus positions are included.

With this, the focus position varying range is determined such that therange includes the focus positions at which the pre-captured imageshaving sharpness levels higher than at least a predetermined thresholdhave been captured. More specifically, since the focus positionscorresponding to the objects having higher sharpness levels can beobtained, a range, in the scene to be captured, in which an image can becaptured with a higher sharpness level can be set as the focus positionvarying range. As a result, a sharp image can be obtained.

Here, it is preferable that the focus varying range determining unit beconfigured to determine the focus position varying range by determining,among the obtained focus positions, a nearest position as a start pointand a farthest position as an end point.

Here, it is also possible that the focus varying range determining unitis configured to determine a plurality of focus position varying rangesby determining, in each of focus position groups, a nearest position asa start point and a farthest position as an end point, the focusposition groups each including, among the obtained focus positions,focus positions satisfying predetermined consecutiveness.

Here, it is preferable that the image capturing unit be configured to:adjust an exposure time based on a size of the determined focus positionvarying range so as to prevent saturation of pixels of an image to becaptured; and capture the imaging target within the adjusted exposuretime while varying the focus position according to the focus positionvarying range.

With this, the exposure time is adjusted according to the size of thefocus position varying range so as to prevent pixel saturation, therebyallowing capturing of the imaging target with as many light signals aspossible. In other words, a shape, high-quality, and clear image can beobtained.

Here, it is preferable that the focus position varying range be limitedby a predetermined exposure time.

With this, the focus position varying range is limited according to apredetermined exposure time, and thus, it is effective when the amountof light is large, for example.

Here, it is preferable that: the image capturing device further include(i) an object detecting unit configured to detect an object in theimaging target; and (ii) an object region identifying unit configured toidentify, based on the detected object, an object region that is apartial region of an image obtained by capturing the imaging target suchthat the detected object is at the center of the image; and that thefocus varying range determining unit be configured to determine thefocus position varying range such that a signal-to-noise ratio of thedetected object is maximized, based on the object region identified bythe object region identifying unit.

With this, identification of the object region enables evaluation ofpre-captured images focusing on the object region, thereby allowing thefocus varying range determining unit to determine the focus positionvarying range. This decreases the data amount required for theevaluation, thereby reducing the time required for determining the focusposition varying range.

Here, it is preferable that: the image capturing device further includea selection receiving unit configured to receive, from a user, selectionof a region of the imaging target in which an object is to be captured;and the object region identifying unit be configured to identify theobject region based on the selection of the region received by theselection receiving unit.

With this, an image with a high sharpness level for at least the objectof the user's interest can be obtained because the object region isrecognized based on information given by the user.

Furthermore, the present invention can, not only be realized as such animage capturing device, but also as an image capturing method to beimplemented in an image capturing device. The present invention can alsobe realized as a program which causes a computer to execute the imagecapturing method. Such a program can be distributed via a recordingmedium such as a CD-ROM or a transmission medium such as the Internet.Furthermore, the present invention can be realized as an integratedcircuit which performs processing of each processing unit.

Advantageous Effects of Invention

An advantageous effect of the present invention is that a focus positionvarying range is automatically determined according to a scene prior toactual image capturing. It is unnecessary for the user to manuallyadjust the focus position varying range. This allows the user to easilyoperate the device with less confusion. The present invention attemptsto maximize high frequency components and signal-to-noise ratio of anobject in an image and appropriately determines a focus position varyingrange based on the scene.

Therefore, a high-quality, sharp, and clear image having an extendeddepth of field can be obtained for various scene types even when thescene to be captured is unknown, and thus the present invention has aneffect which is advantageous over the conventional technique disclosedin Patent Literature 2 and other known methods.

Another advantageous effect of the present invention is that fewer highresolution images are used and stored. The present invention thereforerequires fewer memories than the conventional technique disclosed inPatent Literature 1. Moreover, the present invention has shorter timelags and is capable for real-time applications.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 shows an image capturing method according to a conventionaltechnique.

FIG. 2 is a flow diagram of a conventional method for constructing animage with an extended depth of field.

FIG. 3 is a flow diagram of another conventional method implemented byan image capturing device for increasing the depth of field.

FIG. 4A is an illustrative example of two different capturing scenariosof one scene.

FIG. 4B is an illustrative example of two different capturing scenariosof one scene.

FIG. 5A shows an image capturing device according to an embodiment ofthe present invention.

FIG. 5B shows an image capturing device according to an embodiment ofthe present invention.

FIG. 6A is a block diagram of a device for extending the depth of fieldaccording to an embodiment of the present invention.

FIG. 6B is a flowchart of a depth of field extending process accordingto an embodiment of the present invention.

FIG. 7 is a flowchart of a process for evaluating object blur and/orsharpness level according to an embodiment of the present invention.

FIG. 8 is an illustrative example of images pre-captured at differentfocus positions.

FIG. 9 shows in-focus representation images of the pre-captured imagesin FIG. 8.

FIG. 10 is a flowchart of a process for determining a focus positionvarying range according to an embodiment of the present invention.

FIG. 11A shows an in-focus level of an object in images having differentfocus positions.

FIG. 11B shows an in-focus level of an object in images having differentfocus positions.

FIG. 11C shows an in-focus level of an object in images having differentfocus positions.

FIG. 12 is an illustrative example of an image capturing deviceaccording to an embodiment of the present invention.

FIG. 13 shows an image capturing device according to an embodiment ofthe present invention.

FIG. 14 shows an image capturing device according to another embodimentof the present invention.

FIG. 15 shows an image capturing device according to another embodimentof the present invention.

FIG. 16A shows integrations of high frequency components of objectswithin a given exposure time of different focus sweep ranges.

FIG. 16B shows integrations of high frequency components of objectswithin a given exposure time of different focus sweep ranges.

FIG. 16C shows integrations of high frequency components of objectswithin a given exposure time of different focus sweep ranges.

DETAILED DESCRIPTION OF INVENTION

Hereinafter, embodiments of the present invention will be described indetail with accompanying drawings. Note that the embodiments describedbelow are preferable, specific examples of the present invention. Thestructural elements, the arrangement and connection of the structuralelements, the operation orders, and so on shown in the followingembodiments are given by way of example and are not intended to limitthe present invention. The present invention is limited only by theclaims. Therefore, among the structural elements described in theembodiments below, those not recited in the independent claimsindicating the most generic concept of the present invention are notessential for achieving the objects of the present invention but aredescribed as preferable structural elements.

The present invention provides an image capturing device and an imagecapturing method for extending the depth of field of an image of ascene. In the present invention, at least one image is captured whilethe focus position is varied. A focus position varying range isautomatically and adaptively determined based on the scene to becaptured. The scene is then captured while the focus position is variedaccording to the determined focus position varying range, and at leastone output image is outputted. According to the present invention, thefocus position varying range is determined to maximize thesignal-to-noise ratio of an object in the output image. Therefore, ahigh-quality, sharp, and clear image having an extended depth of fieldcan be obtained for various scene types.

FIG. 4A shows an example of a scene to be captured as an imaging targetin which there are many objects. The relatively near object is close tothe image capturing device and the relatively far object is distant fromthe image capturing device, and there are a plurality of objects inbetween. In order to obtain an image having a large depth of field forall objects, image capturing is performed while the focus position ofthe image capturing device is varied from the near objects to the farobjects, so that high frequency components of all objects are embeddedinto the captured image. Here, for example, the “focus position varyingrange” for this scene is set to SR1, starting from a focus position atwhich the object nearest to the image capturing device is in focus, to afocus position at which the object farthest to the image capturingdevice is in focus.

FIG. 4B shows another example of capturing a scene in which there areonly near objects. The focus position varying range for this sceneshould be adaptively determined to be different to the focus positionvarying range SR1 shown in FIG. 4A. For example, the focus positionvarying range is set to a focus position varying range SR2, which issmaller than SR1. With the focus position varying range SR2, the objectsshown in FIG. 4B are closer to the image capturing device than theobjects shown in FIG. 4A. In other words, the object farthest from theimage capturing device in FIG. 4B is closer to the image capturingdevice than the object farthest from the image capturing device in FIG.4A is. The focus position varying range is changed according to thedistribution of the positions of the objects in the scene relative tothe image capturing device, so as to include at least the focus positionof each object, and thus useful information (high frequency components)of the objects is embedded into the captured image.

Conversely, in the case where the focus position varying range is fixedor predefined, unnecessary information (such as blur) is sometimescaptured. For example, in the case where the focus position varyingrange is predefined as SR1, it may not be suitable to use SR1 for thescene in FIG. 4B. This is because after the focus position is moved passa first object (person) and a second object (trees), the image capturingcontinues even though there are no other objects to be captured infocus, and only blur information of the first and second objects iscaptured when the focus position is varied beyond the focus positionvarying range SR2. As described above, in the case where the blurinformation of the first and second objects is captured, the imagequality and object sharpness level of the output image may be affected.On the other hand, in the case where the focus position varying rangeSR2 is used for the scene in FIG. 4B, the most useful information (highfrequency components) of the objects is captured, and blur informationof the objects is captured in as little amount as possible. Thus, in thecase of capturing the scene in FIG. 4B at a given exposure time, theimage captured using the focus position varying range SR2 has a highersignal-to-noise ratio than the image captured using the focus positionvarying range SR1. As described, because the focus position varyingrange SR2 is smaller, varying of the focus position can be slower withina given capturing time. This allows a longer exposure time.

By adaptively changing the focus position varying range based on thescene, a high-quality output image can be obtained for various scenetypes. The present invention makes it possible to vary the focusposition by moving an imaging lens or an image sensor or both. FIG. 5Aand FIG. 5B show an image capturing device 600 according to anembodiment of the present invention. The image capturing device 600includes an image sensor 6 (12), imaging lenses 8 (14), and an actuator10 (16). The actuator 10 (16) moves the image sensor 6 (12) or theimaging lenses 8 (14) or both. The actuator 10 (16) is realized by alinear motor, a step motor, a servo controlled system, a piezoelectricelement, or a micro electro mechanical system (MEMS). It is preferablethat the actuator 10 (16) be a high-speed and high-accuracy actuator.The image capturing device 600 according to an embodiment of the presentinvention directs the actuator 10 (16) to move at least the image sensor6 (12) or the imaging lenses 8 (14) with a large range SF1 or a rangeSF2 smaller than SF1 according to the scene, so as to capture an imagewhile changing the focus position varying range according to the scene.Note that the focus position varying range is automatically determined.The details will be described next.

FIG. 6A is a block diagram of the image capturing device 600 forextending the depth of field of an image according to an embodiment ofthe present invention. The image capturing device 600 includes apre-capturing module 602, an object sharpness evaluating module 604, afocus varying range determining module 606, an image capturing module608, a deconvoluting module 610, and an internal buffer 612.

The pre-capturing module 602, functioning as a pre-capturing unit,captures a scene that is an imaging target at each of different focuspositions, and outputs a plurality of pre-captured images which arelower in resolution than an output image to be finally outputted by theimage capturing device 600 (see below). The pre-captured imagesoutputted by the pre-capturing module 602 may be stored in the internalbuffer 612 or be directly processed by the object sharpness evaluatingmodule 604.

The object sharpness evaluating module 604 evaluates object sharpnesslevels of the plurality of pre-captured images. Note that the objectsharpness evaluating module 604 may evaluate object blur instead ofobject sharpness level. Here, the object sharpness evaluating module 604may evaluate the object sharpness level (or blur) such that the objectsharpness level is higher as spatial frequency in a pre-captured imageis higher, or evaluate that the object sharpness level is higher as thecontrast is greater. This object sharpness evaluation results inindication of object in-focus levels of the pre-captured images. Thisevaluation is hereinafter described as “object in-focus level”, but itis interchangeable with “object sharpness level” and “object blur”. Notethat in the case of interchanging with “object blur”, the magnitude ofthe “object in-focus level” should be reversed. The evaluation resultmay be stored in the internal buffer 612 or be directly processed by thefocus varying range determining module 606.

The focus varying range determining module 606 appropriately determinesa focus position varying range suitable for each scene to be captured.More specifically, the focus varying range determining module 606determines a focus position varying range that maximizes asignal-to-noise ratio of objects in an output image of the scene to befinally outputted by the image capturing device 600. This determinationis based on the object sharpness (or blur) evaluation result.

Note that the object sharpness evaluating module 604 and the focusvarying range determining module 606 function as a focus varying rangedetermining unit, and determine a focus position varying range based onthe sharpness levels of the pre-captured images captured by thepre-capturing module 602 functioning as the pre-capturing unit, suchthat the signal-to-noise ratio of the output image is maximized.

The image capturing module 608, functioning as an image capturing unit,captures an imaging target while varying the focus position according tothe focus position varying range determined by the object sharpnessevaluating module 604 and the focus varying range determining module 606functioning as the focus varying range determining unit, and outputs atleast one output image. When capturing a scene that is an imagingtarget, the image capturing module 608 simultaneously starts varying ofthe focus position and capturing of at least one image. The focusposition varied at this time is based on the focus position varyingrange determined. This results in generation of a convoluted imagecorresponding to the focus position varying range.

The deconvoluting module 610, functioning as an extended depth of fieldimage deriving unit, derives an image having an extended depth of field,using at least one image outputted by the image capturing unit and atleast one predetermined point spread function (PSF). More specifically,the deconvoluting module 610 uses at least one predetermined PSF todeconvolute a captured image (convoluted image) outputted by the imagecapturing module 608, to derive an image having an extended depth offield. The image having an extended depth of field outputted by thedeconvoluting module 610 is also called an “all in-focus (AIF)” image,and may be used for viewing purpose. Alternatively, the AIF image may beused in further processing modules such as depth estimation and depthimage based rendering (DIBR). Further details will be described in otherembodiments.

The pre-capturing module 602, the object sharpness evaluating module604, the focus varying range determining module 606, the image capturingmodule 608, the deconvoluting module 610, and the internal buffer 612included in the image capturing device 600 are typically achieved in theform of integrated circuits (IC), application-specific integratedcircuits (ASIC), large scale integrated (LSI) circuits, a digital signalprocessor (DSP), or achieved by a CPU based processor and machineincluding a personal computer (PC). Each of these modules may be in aplurality of single-function LSIs or one integrated LSI. The name usedhere is LSI, but it is also called IC, system LSI, super LSI, or ultraLSI in accordance with the degree of integration. Moreover, ways toachieve integration are not only LSI, but a special circuit or a generalpurpose processor and so forth can also achieve the integration. Thisincludes a specialized microprocessor such as a digital signal processor(DSP) that can be directed by program instruction. A field programmablegate array (FPGA) that can be programmed after manufacturing LSI or areconfigurable processor that allows reconfiguration of the connectionor configuration of LSI can be used for the same purpose. In the future,with advancement in manufacturing and process technology, a brand-newtechnology may replace LSI. The integration can be achieved by thattechnology. In the implementation, the image capturing device 600 may beembedded into an image capturing device such as a digital still cameraand a movie camera. The image capturing device 600 may also beimplemented in a standalone device to work with an image capturingsystem in a professional capturing system, for example. Theimplementation of the image capturing device 600 in other types ofdevice is also possible, which does not limit the scope of the presentinvention.

FIG. 6B is a flowchart of a depth of field extending process accordingto an embodiment of the present invention.

First at step S602, the pre-capturing module 602 captures (pre-captures)a scene that is an imaging target at each of different focus positionsto output a plurality of pre-captured images. The term “pre-capture” isused here which refers to the process for capturing the scene forpre-computing a focus position varying range (hereinafter referred to as“pre-capturing process”). The images captured in the pre-capturingprocess are not directly used for constructing an output image. Thepre-capturing process may be performed while the camera is in previewmode. For example, for a movie camera, the preview mode is on while theuser is pointing the camera to the intended scene. The pre-capturingprocess may be performed just before the record button is pressed.Typically, the number of images captured in the pre-capturing process isless than 10. This means that the pre-capturing process requires only330 milliseconds to finish capturing all 10 images, at a normal videorate (30 frames per second). This delay is acceptable because thepre-capturing process is needed only one time for the same scene.Another example is the case of a digital still camera. In this case, thepre-capturing process may be performed just after the user presses theshutter button halfway. This does not cause a delay noticeable to theuser.

There are other known processing methods for the above pre-capturingprocess. Examples include the pre-capturing process performed for theexposure and shutter speed determination function and the autofocusfunction. It is possible that the present invention utilizes suchexisting pre-capturing process. Specifically, the pre-capturing processfor the autofocus function may be directly used for the presentinvention. This is because the pre-captured images obtained from thepre-capturing process according to an aspect of the present inventionmay have different focus points. Other pre-capturing process may also beused. This is not to limit the scope and spirit of the presentinvention. It is preferable that the resolutions of the pre-capturedimages obtained from the pre-capturing process be relatively low toallow efficiency in memory storage and in further processing.

The focus position of each image in the pre-capturing process may beeffectively varied by directing the actuator to move the image sensor orthe imaging lenses or both.

At step S604, the object sharpness evaluating module 604 evaluatesobject sharpness levels (or blur) of the pre-captured images to obtainimages indicating object in-focus levels of the pre-captured images.When an image is pre-captured in the pre-capturing process, the focusposition of the image is recorded. Objects at different distances fromthe image capturing device in the scene appear sharply in-focus indifferent pre-captured images. For a specific object, it has the maximumsharpness level (maximum in-focus level) in only one pre-captured image.Therefore, by checking the focus position corresponding to apre-captured image having the maximum sharpness level of an object, thedistance from that object to the image capturing device (hereinafterreferred to as “object distance”) can be computed. Moreover, since anobject distance can be computed for each object, computation of relativedistances from an object to other objects is also possible.

At step S606, the focus varying range determining module 606 determinesa focus position varying range based on the pre-captured images obtainedfrom step S604, such that a signal-to-noise ratio (SNR) of objects inthe image of a scene to be captured is maximized. For example, in oneembodiment, the start point and the end point of the focus positionvarying range may be determined using at least two object distancesderived from the highest in-focus levels of objects in the imagesobtained from step S604. The start point and the end point determined inthis case correspond to the minimum object distance and the maximumobject distance in the scene, respectively. More specifically, the startpoint and the end point are determined such that one start point iscorresponded to the minimum object distance in the scene and one endpoint is corresponded to a distance within a range between the minimumand maximum object distances in the scene. That is to say, the focusvarying range determining module 606 determines a focus position varyingrange by determining, among the focus positions obtained at step S604,the nearest position as the start point and the farthest position as theend point. Note that the focus varying range determining module 606 maydetermine a plurality of focus position varying ranges by determining,as the start and end points respectively, the nearest and farthestpositions in each of focus position groups which includes, among thefocus positions obtained from step S604, a plurality of focus positionssatisfying predetermined consecutiveness. In this case, the “focuspositions satisfying predetermined consecutiveness” refer to focuspositions corresponding to consecutively pre-captured images among, forexample, 10 pre-captured images. For instance, in the case where thefocus positions obtained from step S604 correspond to the first to thirdpre-captured images and the seventh to tenth pre-captured images, twofocus position ranges are determined as focus position varying ranges;one being a range between the focus positions corresponding to the firstto third pre-captured images, and the other being a range between thefocus positions corresponding to the seventh to tenth pre-capturedimages.

At step S608, the image capturing module 608 captures a scene that is animaging target while varying the focus position during image capturingbased on the focus position varying range determined at step S606, andoutputs at least one output image. The light signals of objects at bothin-focus and out-of-focus positions in the scene are accumulated intopixels in the image sensor 6 (12), forming an output image (convolutedimage). Note that in the case of more than one focus position varyingrange, the focus position varying speed may be set faster to reduce theexposure time in ranges outside the focus position varying ranges, orimages may be captured in each of the focus position varying ranges. Inthe latter case, the number of images captured is the number of focusposition varying ranges.

At step S610, the deconvoluting module 610 deconvolutes the imagecaptured (convoluted) at step S608, using at least one point spreadfunction (PSF), to obtain an image having an extended depth of field.Note that in the case where a plurality of focus position varying rangesare set and a plurality of images are thus captured, an image which isthe plurality of images convoluted is deconvoluted using a point spreadfunction corresponding to that image to obtain an image having anextended depth of field.

By adaptively determining the focus position varying range based on anobject distance in the scene according to the above depth of fieldextending process, the light signals of all objects at in-focuspositions are accumulated (integrated) into pixels. This is to guaranteethat high frequency components (referred to the light signals of objectsat in-focus positions) of objects in the image of a scene are maximized.Thus, a high-quality image having an extended depth of field can beobtained for various scene types.

Note that a known deconvolution method may be used at step S610. Oneexample of such deconvolution method is “Wiener deconvolution” or“Wiener filter”. The details about the method can be found in “DigitalImage Processing” of R. C. Gonzalez & R. E. Woods, Addison-WesleyPublishing Company, Inc., 1992 (Non-Patent Literature 2). Anotherexample of the known method is “Lucy-Richardson deconvolution”. Thedetails about the method can be found in “Acceleration of IterativeImage Restoration Algorithms” of D. S. C. Biggs, Applied Optics, Vol.36, No. 8, pp. 1766-1775, 1997 (Non-Patent Literature 3).

The PSF used at step S610 is stored in a memory in advance. The PSFstored in the memory in advance may be pre-computed by calibrating theimage capturing device 600. For example, a calibration chart is placedin the scene and captured at different focus positions. The distancebetween the calibration chart and the image capturing device isadaptively changed when the calibration chart is captured, and a PSF isthen computed using calibration data obtained from the calibration chartcapturing. Alternatively, a PSF may be computed when an image iscaptured, by employing a PSF estimation technique. In such a manner,deconvolution such as a blind deconvolution technique is used at stepS610.

FIG. 7 is a detailed diagram of an object sharpness evaluation methodaccording to an exemplarily embodiment of the present invention.Specifically, FIG. 7 explains the details of the process at step S604 inthe depth of field extending process.

This method starts when the pre-captured images captured at differentfocus positions at step S602 are outputted. First, at step S702, theobject sharpness evaluating module 604 takes in the pre-captured imagesas input images and smoothes each of the input images.

At step S704, the object sharpness evaluating module 604 computes avertical gradient image and a horizontal gradient image of an inputimage and computes a vertical gradient image and a horizontal gradientimage of a smoothed input image. At step S706, the object sharpnessevaluating module 604 computes a difference image between the verticalgradient images of the input image and the smoothed image, and computesa difference image between the horizontal gradient images of the inputimage and the smoothed input image. At step S708, the object sharpnessevaluating module 604 computes for each of the input images(pre-captured images) an in-focus representation image indicating thein-focus level of the corresponding input image, using the computeddifference image of the vertical gradient and the computed differenceimage of the horizontal gradient, to obtain a resultant image of thisstep. At step S710, the method stores the focus position and thein-focus representation image computed for the corresponding inputimage, resulting in indication of the object distance and the objectin-focus level of the image. Steps S702 to S710 are repeated until allpre-captured images are evaluated.

FIG. 8 shows an example of pre-captured images captured at differentfocus positions in the pre-capturing process. For example, the scene iscaptured at different focus positions, starting from a near focusposition to a far focus position. In FIG. 8, an image 801 is captured atthe near focus position. The focus point of the image 801 is set at thenear object. The focus position is varied with time. An image 805 showsthat the focus position is changed to the intermediate position, and thefocus point is at the intermediate object. The focus position is thenchanged to a far focus point. At the far focus position, the focus pointis set at the far object as shown in an image 809. After thepre-capturing process, pre-captured images each having a different focuspoint are obtained.

FIG. 9 shows a real example of in-focus level distribution of each ofthe pre-captured images. These are 10 pre-captured images. An image A isan image captured when the focus position is set at near focus. Thisfocus position is varied from near focus to far focus. An image J is animage captured when the focus position is set at far focus. In FIG. 9,it is shown that the in-focus level is increasing from the image A to animage E. This means that in the image E the nearest object is in-focus.The in-focus level of this image is high and decreasing in an image J.It means that the farthest object is blurring (out of focus) because thein-focus levels of all objects are decreasing. The highest in-focuslevel of a far object is found in an image I. Therefore, the focuspositions of the images E and I can be used for determining a focusposition varying range.

FIG. 10 is a detailed diagram of a focus varying range determiningprocess according to an embodiment of the present invention.Specifically, FIG. 10 explains the details of the process at step S606in the depth of field extending process.

This method starts when the object in-focus levels are indicated at stepS604. First, at step S1002, the focus varying range determining module606 obtains a plurality of in-focus level distributions and a pluralityof focus positions corresponding to the plurality of input images. Atstep S1004, the focus varying range determining module 606 identifies afirst image having a maximum in-focus level of the nearest object in ascene and identifies a second image having a maximum in-focus level ofthe farthest object in the scene. At step S1006, the focus varying rangedetermining module 606 computes a focus position varying range based onthe focus positions of the identified first and second images such thatthe signal-to-noise ratio of an output image to be finally outputted ismaximized.

FIG. 11A to FIG. 11C show examples of in-focus levels of objects in ascene across different focus positions. Note that each focus positionshown in FIG. 11A to FIG. 11C represents a nearer position when thenumber is smaller and a farther position when the number is larger.Specifically, FIG. 11A shows an in-focus level across different focuspositions in the case of capturing a first object 811 in FIG. 8. In FIG.11A, the in-focus level of the object 811 increases from a focusposition 1 to a focus position 5 and decreases successively thereafter.

FIG. 11B shows an in-focus level across different focus positions in thecase of capturing a second object 812 in FIG. 8. In FIG. 11B, thein-focus level of the object 812 increases from the focus position 1 toa focus position 6, horizontally transits to focus positions 7 and 8,and decreases thereafter. This indicates that FIG. 11B shows thein-focus level corresponding to the intermediate object.

FIG. 11C shows an in-focus level across different focus positions in thecase of capturing a third object 813 in FIG. 8. In FIG. 11C, thein-focus level of the object 813 increases from the focus position 1 toa focus position 9, and decreases thereafter. This indicates that FIG.11C shows the in-focus level corresponding to the far object.

From these examples, the maximum in-focus level of the first object 811,the nearest object, is identified as P5 and the maximum in-focus levelof the farthest object is identified as P9. Therefore, a focus positionvarying range is derived from the focus positions of the pre-capturedimages corresponding to P5 and P9.

With this, the focus varying range determining module 606 obtains, foreach pre-captured image, an in-focus level (sharpness level) of each ofregions obtained by dividing each pre-captured image, and obtains, foreach region, a focus position corresponding to a pre-captured imagehaving a maximum sharpness level among the in-focus levels (sharpnesslevels) of the regions obtained for each pre-captured image. The focusvarying range determining module 606 then determines the focus positionvarying range such that the obtained focus positions are included.

Thus, the focus varying range determining module 606 obtains for eachregion a focus position corresponding to the pre-captured image havingthe maximum in-focus level (sharpness level), and determines a focusposition varying range based on the focus positions obtained. With this,in the case where an object is present in a region obtained by dividingeach pre-captured image, the focus position corresponding to that objectcan be obtained. In other words, because a focus position varying rangeis determined such that the obtained focus positions are included, it ispossible to obtain an image in which each object included in the sceneto be captured appears sharply.

Furthermore, the focus varying range determining module 606 may: obtain,for each pre-captured image, an in-focus level (sharpness level) of eachof regions obtained by dividing each pre-captured image; obtain aplurality of focus positions at which pre-captured images having, amongthe obtained in-focus levels of the pre-captured images and the regions,sharpness levels higher than a predetermined threshold have beencaptured; and determine the focus position varying range such that theobtained focus positions are included.

With this, the focus varying range determining module obtains, for eachpre-captured image, an in-focus level (sharpness level) of each ofregions obtained by dividing each pre-captured image, and obtains aplurality of focus positions at which pre-captured images having, amongthe in-focus levels (sharpness levels) obtained for each pre-capturedimage and each region, in-focus levels (sharpness levels) higher than apredetermined threshold have been captured. Then, the focus varyingrange determining module determines a focus position varying range suchthat the obtained focus positions are included.

With this, the focus position varying range is determined such that therange includes the focus positions at which the pre-captured imageshaving in-focus levels (sharpness levels) higher than at least apredetermined threshold have been captured. More specifically, since thefocus positions corresponding to the objects having higher in-focuslevels (sharpness levels) can be obtained, a range, in the scene to becaptured, in which an image can be captured with a higher in-focus level(sharpness level) can be set as the focus position varying range. As aresult, a sharp image can be obtained.

FIG. 12 is a schematic block diagram of hardware of an image capturingdevice 600 according to an embodiment of the present invention. Theimage capturing device 600 includes an optical system 1202, an imagesensor 1204, an analog-to-digital converter (ADC) 1206, an imageprocessor 1208, a microcomputer 1210, an external memory 1212, a drivercontroller 1220, an optical image stabilizer (OIS) sensor 1218, anoperating unit 1222, a storing and/or transmitting device 1216, and adisplay device 1214. The image processor 1208 includes an internalmemory 1240, an extended depth of field module 1246, a raw imageprocessor 1242, a color image processor 1243, and optionally a 3D imageprocessor 1244. Other structural elements such as a microphone, aspeaker, and so on are not shown, but this does not limit the scope andspirit of the present invention.

The optical system 1202 may include structural elements such as lensesor a set of lenses, a zoom/focus mechanism, an actuator, a shutter, anaperture, and so on, for controlling the light signal reaching the imagesensor 1204.

The image sensor 1204 accumulates the incoming light collected by theoptical system 1202 and converts the light into an electrical signal.The image sensor 1204 is controlled by the microcomputer 1210. Theelectrical signal generated by the image sensor 1204 is converted intodigital data (raw image data) by the ADC 1206 and stored in the internalmemory 1240 or the external memory 1212. The raw image data includes aset of pre-captured images each of which is taken at a different focusposition. In addition, the raw image data includes an image higher inresolution than the pre-captured images. The image higher in resolutionthan the pre-captured images is the raw image data convoluted withimages captured at different focus positions while the focus position isvaried during image capturing.

The raw image processor 1242 takes in the raw image data from theinternal memory 1240 (or the external memory 1212) and may perform manypre-processes (not shown), such as resizing, linearity correction, whitebalance, and gamma correction. This pre-processed raw image data may bestored or transmitted by the storing and/or transmitting device 1216.The pre-processed raw image data can also be processed by the colorimage processor 1243 to generate a color image, such as RGB or YCbCr.The color image processor 1243 performs such processing as colorinterpolation, color correction, tonal range adjustment, and color noisereduction, for generating a favorable color image. The extended depth offield module 1246 takes in the pre-captured images and directs themicrocomputer 1210 to drive the imaging lenses or image sensor forvarying the focus position during image capturing. The captured image isthen inputted to the extended depth of field module 1246 and an extendeddepth of field image (referred to as “all-in-focus (AIF) image”) isgenerated. The output AIF image may be used for viewing on the displaydevice 1214 and may also be stored in the storing and/or transmittingdevice 1216. Examples of a storage device in the storing and/ortransmitting device 1216 include, but are not limited to, a flash-basedmemory card, a hard drive, and an optical drive. Examples of atransmission device in the storing and/or transmitting device 1216include, but are not limited to, an HDMI interface, a USB interface, awireless interface, and a direct-to-printer interface. The dataprocessed by the storing and/or transmitting device 1216 may optionallyundergo lossless or lossy compression. Furthermore, the output AIF imagemay be used in the further processing module such as the 3D imageprocessor 1244. The AIF image may be used for depth estimation and 3Dimage generation. The details thereof will be described in the nextembodiment.

The optical system 1202 may be controlled by the driver controller 1220which is controlled by the microcomputer 1210. The operating unit 1222receives user operation input and sends an electrical signal to themicrocomputer 1210 to control the related modules such as the drivercontroller 1220, the image sensor 1204, and the image processor 1208that correspond to the user input. The OIS sensor 1218 detects a motioncaused by hand shaking or camera motion and sends an electrical signalto the microcomputer 1210. The microcomputer 1210 directs the drivercontroller 1220 to control an actuator or the like in the optical system1202 so that the actuator moves the lenses to compensate for the motionvia the driver controller 1220, thus reducing motion blur effect causedby hand shaking or camera motion.

The image processor 1208 may send a first electrical signal provided bythe extended depth of field module 1246, to the microcomputer 1210. Themicrocomputer 1210 directs the driver controller 1220 to control anactuator or the like in the optical system 1202 so that the focusposition of each image is varied during the pre-capturing process.

The image processor 1208 may send a second electrical signal provided bythe extended depth of field module 1246, to the microcomputer 1210. Themicrocomputer 1210 directs the driver controller 1220 to control anactuator or the like in the optical system 1202 so that the actuatormoves the lenses for motion compensation and the focus position isvaried during the pre-capturing process, thereby forming a convolutedimage to be inputted to the extended depth of field module 1246 forgeneration of an extended depth of field image.

Note that the extended depth of field module 1246 is hardware forachieving the function indicated by the image capturing device 600 inthe previous embodiments.

The image processor 1208, the extended depth of field module 1246, andthe modules therein are typically achieved in the form of integratedcircuits (IC), application-specific integrated circuits (ASIC), or largescale integrated (LSI) circuits. Each of these modules may be in aplurality of single-function LSIs or in one integrated LSI. The nameused here is LSI, but it is also called IC, system LSI, super LSI, orultra LSI in accordance with the degree of integration. Moreover, waysto achieve integration are not only LSI, but a special circuit or ageneral purpose processor and so forth can also achieve the integration.This includes a specialized microprocessor such as a digital signalprocessor (DSP) that can be directed by program instruction. A fieldprogrammable gate array (FPGA) that can be programmed aftermanufacturing LSI or a reconfigurable processor that allowsreconfiguration of the connection or configuration of LSI can be used.In the future, with advancement in manufacturing and process technology,a brand-new technology may replace LSI. The integration can be achievedby that technology.

With the image capturing device 600 according to the present embodiment,the pre-capturing module 602 obtains, before the imaging target isactually captured to generate an output image, pre-captured images bycapturing the imaging target plural times, each time with a differentfocus position. The object sharpness evaluating module 604 thenevaluates the sharpness levels of the pre-captured images. Thus, sincethe object sharpness evaluating module 604 evaluates in advance thesharpness levels of the pre-captured images captured at the plural focuspositions, the position, relative to the image capturing device, of anobject in the scene which is the imaging target can be known. With this,the focus varying range determining module 606 can automaticallydetermine a focus position varying range such that the range includesthe focus position of each object at which the object can be capturedsharply in the scene to be captured, for example.

As described, since, prior to actual image capturing, a focus positionvarying range is automatically determined according to the scene to becaptured, it is unnecessary for the user to manually adjust the focusposition varying range to suit the scene. This allows the user to easilyoperate the image capturing device with less confusion about how tooperate the device. Therefore, a high-quality, sharp, and clear imagewith an extended depth of field can be obtained for various scene typeseven when the position, relative to the image capturing device 600, ofan object in the scene to be captured is unknown.

Moreover, the pre-captured images obtained to determine a focus positionvarying range are lower in resolution than an output image, and thusfewer high resolution images need to be used and stored. Therefore,fewer memories are required than the conventional technique disclosed inPatent Literature 1. Moreover, the present invention has shorter timelags and is capable of real-time applications.

FIG. 13 is an exemplarily block diagram of a depth of field extendingdevice 1300 which includes an image capturing device according to anembodiment of the present invention. In this embodiment, the details ofthe depth of field extending device 1300 will be described in theapplication on 3D (left and right) image generation, by way ofillustrative example. The depth of field extending device 1300pre-captures a plurality of images of a scene to be captured, and storesthe pre-captured images in a memory 1340. Each of the pre-capturedimages is taken at a different focus position. Therefore, objectslocated in different pre-captured images each have a sharpness leveldifferent from one another. A controller 1320 directs an actuator (notshown) to move at least one of a lens system 1302 and an image sensor1304 for varying the focus positions of images in the pre-capturingprocess. An object sharpness evaluating module 1346 takes in theplurality of images from the memory 1340 and evaluates the object blurand/or sharpness level of each image to obtain data indicating an objectdistance and an object in-focus level corresponding to an object in theimage. The object sharpness evaluating module 1346 may store theobtained data in the memory 1340, and may also send the obtained data toa focus varying range determining module 1342. The focus varying rangedetermining module 1342 takes in the obtained data and determines afocus position varying range. More specifically, the focus varying rangedetermining module 1342 determines the start point and the end point ofa focus position varying range based on the obtained data by taking intoconsideration the highest in-focus level obtained for a near object andthe highest in-focus level obtained for a far object. The focus varyingrange determining module 1342 derives the start point of the focusposition varying range from the focus position of a pre-captured imagehaving the highest in-focus level of the near object, and derives theend point of the focus position varying range from the focus position ofa pre-captured image having the highest in-focus level of the farobject. This means that the image of the scene to be captured while atleast one of the lens system 1302 and the image sensor 1304 is movedaccording to this focus position varying range contains the highestin-focus levels of all near and far objects. Therefore, the imagecaptured while the focus position is varied within this determined rangeis an image convoluted with images in which all objects appear with themaximum sharpness levels.

Referring to FIG. 13, the focus varying range determining module 1342sends an electrical signal indicating the focus position varying rangeto the controller 1320. The controller 1320 directs the actuator to varythe focus position during image capturing, based on the electricalsignal indicating the focus position varying range. The depth of fieldextending device 1300 captures at least one image while varying thefocus position and stores the captured image in the memory 1340. Adeconvoluting module 1343 takes in the captured image stored in thememory 1340 and deconvolutes that image using at least one predeterminedpoint spread function (PSF) to obtain an AIF image. The depth of fieldextending device 1300 may further capture additional near-focus imageand far-focus image. Optionally, the near-focus and far-focus images maybe used in the deconvoluting module 1343. The AIF image may be used forviewing as an image (in 2D) with an enhanced depth of field. The AIFimage, the near-focus image, and the far-focus image may be used in adepth estimating module 1345. The depth estimating module 1345 estimatesa depth map using Advanced-Depth from Defocus (A-DFD) technique. Theestimated depth map and the AIF image are inputted to a Depth ImageBased Rendering (DIBR) module 1347. The DIBR module 1347 generatesstereoscopic left and right images which can be used for 3D viewing.

In one embodiment, the focus position varying range may be determinedadaptively based on the scene to be captured. For example, the focusposition varying range for capturing the scene containing objects atdistance from 1.0 meter to 2.5 meters may be smaller than the focusposition varying range for capturing the scene containing objects atdistance from 1.0 meter to 10 meters. According to the following thinlens equation:

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack\mspace{515mu}} & \; \\{\frac{1}{f} = {\frac{1}{u} + \frac{1}{v}}} & \left( {{Expression}\mspace{14mu} 1} \right)\end{matrix}$where, in (Expression 1), f is the focal length of the imaging system inthe depth of field extending device 1300, u is the distance between anobject in the scene and the imaging lens, v is the distance between theimaging lens and the image plane in which an object at distance uappears sharply in-focus. For example, when the imaging system having afocal length f=18 mm is focusing on an object at the distance of 1.0meter, the distance between the imaging lens and the image plane inwhich the object appears in-focus can be computed as shown in(Expression 2) and (Expression 3) below.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 2} \right\rbrack\mspace{515mu}} & \; \\{\frac{1}{18} = {\frac{1}{1000} + \frac{1}{v}}} & \left( {{Expression}\mspace{14mu} 2} \right) \\{\left\lbrack {{Math}.\mspace{14mu} 3} \right\rbrack\mspace{515mu}} & \; \\{v = {{1/\left( {\frac{1}{18} - \frac{1}{1000}} \right)} = {18.3299\mspace{14mu}{mm}}}} & \left( {{Expression}\mspace{14mu} 3} \right)\end{matrix}$

When this imaging system is focusing on the object at the distance of2.5 meters, the distance between the imaging lens and the image plane inwhich the object appears in-focus is v=18.1305 mm. Therefore, in thisexample, the focus position varying range for capturing the scenecontaining objects at distance from 1.0 meter to 2.5 meters is set to arange from 18.3299 mm to 18.1305 mm, for example. In this case, theactuator is directed to move the imaging lens or the image sensor withthe distance of 199.4 um, for example. In the case where the aboveimaging system is focusing on the object at the distance of 10.0 meters,the distance between the imaging lens and the image plane in which theobject appears in-focus is v=18.0324 mm. Therefore, in this example, thefocus position varying range for capturing the scene containing objectsat distance from 1.0 meter to 10.0 meters is set to a range from 18.3299mm to 18.0324 mm. In this case, the actuator is directed to move theimaging lens or the image sensor with the distance of 297.5 um, forexample.

By doing this adaptively, the in-focus levels of objects (at focuspoint) in the image captured (while the focus position is varied) aremaximized. In addition, the signal-to-noise ratio of the objects is alsomaximized especially for the image of the scene containing onlyrelatively near objects, because the focus position varying range issmaller.

FIGS. 16A to 16C show computation, through integrations of contrast, ofobject in-focus levels within a given exposure time of different focussweep (varying) ranges. In FIG. 16A, the scene contains an object OB1(relatively close to the camera), an object OB2 (at the center), and anobject OB3 (relatively far from the camera). The focus position varyingrange is set such that maximum in-focus points (maximum contrast) of theobjects OB1, OB2, and OB3 are integrated. The range corresponds to adistance D1. The amount of time for integrating light signals of theobjects at the respective positions is computed as [Integration Time(T)/Distance (D1)], for example. In FIG. 16B, the scene contains onlythe object OB1 (relatively close to the camera) and the object OB2 (atthe center). When the focus position varying range is predefined as isconventionally, it is set similarly to D1, which is however not proper.As shown in FIG. 16B, the amount of time for integrating the lightsignals of the objects at the respective positions is computed as[Integration Time (T)/Distance (D2)], and D2=D1. This however is notproper because there is no object after the object OB2. It isunnecessary to include such range having no object after the object OB2into the focus position varying range. In FIG. 16C, the scene containsonly the object OB1 (relatively close to the camera) and the object OB2(at the center) as in the case of FIG. 16B. However, the focus positionvarying range is changed and is smaller. This is because only the lightsignals from the highest in-focus position of the object OB1 to thehighest in-focus position of the object OB2 are integrated. The amountof time for integrating the light signals of the objects at therespective positions is [Integration Time (T)/Distance(D3)]>[Integration Time (T)/Distance (D2)]. More specifically, in thecase of capturing the scene including only the objects OB1 and OB2, aplurality of images are pre-captured to obtain object contrastscorresponding to different focal lengths, and thus a range excluding anobject distance beyond the object OB2 is automatically set as the focusposition varying range. As described, since the focus position varyingrange can be set according to the range in which objects are present,the signal-to-noise ratio at the in-focus positions of the objects inthe captured image is maximized.

In one embodiment, the focus position varying range is limited by themovement speed of the actuator. For example, when the maximum movementspeed of the actuator is 43 mm/second, the maximum range limit for videoframe rate (30 frames per second) is 43/30=1.433 mm.

In one embodiment, the exposure time of the imaging system may beadjusted based on the determined focus position varying range. This iseffective for still pictures. For example, when the initial exposuretime is set as ⅛ seconds, the determined range is 4.3 mm, and theactuator moves with the maximum speed of 43 mm/second, the exposure timefor this range may be adjusted to 1/10 seconds. On the other hand, themovement speed of the actuator may be adjusted to match with theexposure time. In this example, the movement speed of the actuator maybe adjusted to 34.4 mm/second.

Moreover, the exposure time may be adjusted according to the size of thefocus position varying range so as to prevent pixel saturation. In thiscase, an imaging target can be captured with as many light signals aspossible. In other words, a shape, high-quality, and clear image can beobtained. Furthermore, the focus position varying range may be limitedaccording to a predetermined exposure time. This is effective when theamount of light is large, for example.

By capturing the image of a scene while varying the focus positionduring image capturing according to an aspect of the present invention,the image quality of the output AIF image is improved. When the outputAIF image is used for viewing (2D viewing), sharper and clearer objectscan be seen. When the output AIF image is used for depth estimation(1345), the accuracy of the estimated depth map is improved. When theoutput AIF image is used for 3D image generation, a sharp and clear 3Dimage can be obtained and the 3D viewing effect is improved.

FIG. 14 is a block diagram of an image capturing device 1400 accordingto another embodiment of the present invention. The difference betweenthis embodiment and the embodiment in FIG. 13 is that an objectdetecting module 1446 is used. The object detecting module 1446,functioning as an object detecting unit, detects an object such as facein a scene to be captured as an imaging target. Furthermore, the objectdetecting module 1446, functioning also as an object region identifyingunit, identifies, based on the detected object such as face, a range ofinterest (ROI) as an object region that is a partial region of an imageobtained by capturing the imaging target in a way that the detectedobject is at the center of the image. ROI data related to the identifiedrange of interest may be sent to a focus varying range determiningmodule 1442. The focus varying range determining module 1442 determinesthe focus position varying range based on the ROI data such that thesignal-to-noise ratio of the objects detected in the image of the sceneto be captured is maximized. The focus varying range determining module1442 sends an electrical signal indicating the focus position varyingrange to a controller 1420. The controller 1420 directs the actuator tovary the focus position during image capturing. The image capturingdevice 1400 captures at least one image while varying the focus positionand stores the image in a memory 1424. A deconvoluting module 1443 takesin and deconvolutes the captured image to obtain an output AIF image.The image capturing device 1400 may further capture additionalnear-focus image and far-focus image. Optionally, the near-focus andfar-focus images may be used in the deconvoluting module 1443. The AIFimage may be used for viewing as an image (in 2D) with an enhanced depthof field. The AIF image, the near-focus image, and the far-focus imagemay be used in a depth estimating module 1445. The depth estimatingmodule 1445 estimates a depth map using Advanced-Depth from Defocus(A-DFD) technique. The estimated depth map and the AIF image areinputted to a Depth Image Based Rendering (DIBR) module 1447. The DIBRmodule 1447 generates stereoscopic left and right images which can beused for 3D viewing.

As described, identification of the object region as the range ofinterest (ROI) enables evaluation of pre-captured images focusing on theROI, thereby allowing the focus varying range determining module 1442 todetermine the focus position varying range. This decreases the dataamount required for the evaluation, thereby reducing the time requiredfor determining the focus position varying range.

FIG. 15 is a block diagram of an image capturing device 1500 accordingto another embodiment of the present invention. The difference betweenthis embodiment and the embodiment in FIG. 13 is that a user interfacemodule 1546 is incorporated. The user interface module 1546, functioningas a selection receiving unit, receives, as an object of interest (OOI),user's selection of a region of an imaging target in which an object isto be captured. Such selection is given via a user interaction devicesuch as a touch screen display device provided in a display module 1514.The user interface module 1546, also functioning as an object regionidentifying unit, identifies a range of interest (ROI) in the scenebased on OOI data related to the received OOI, such that the OOI is atthe center of the ROI. ROI data related to the identified ROI is sent toa focus varying range determining module 1542. The focus varying rangedetermining module 1542 determines a focus position varying range basedon the ROI data, such that a signal-to-noise ratio of the OOI detectedin the image of the scene to be captured is maximized. The focus varyingrange determining module 1542 sends an electrical signal indicating thefocus position varying range to a controller 1520. The controller 1520directs the actuator to vary the focus position during image capturing.The image capturing device 1500 captures at least one image whilevarying the focus position and stores the image in a memory 1524. Adeconvoluting module 1543 takes in and deconvolutes the captured imageto obtain an output AIF image. The image capturing device 1500 mayfurther capture additional near-focus image and far-focus image.Optionally, the near-focus and far-focus images may be used in thedeconvoluting module 1543. The AIF image may be used for viewing as animage (in 2D) with an enhanced depth of field. The AIF image, thenear-focus image, and the far-focus image may be used in a depthestimating module 1545. The depth estimating module 1545 estimates adepth map using Advanced-Depth from Defocus (A-DFD) technique. Theestimated depth map and the AIF image are inputted to a Depth ImageBased Rendering (DIBR) module 1547. The DIBR module 1547 generatesstereoscopic left and right images which can be used for 3D viewing.

With this, an image with a high sharpness level for at least the objectof the user's interest can be obtained because the object region (rangeof interest) is recognized based on the object of interest (OOI) that isinformation given by the user.

The present invention is useful as an image capturing device and so onthat is efficient, uses less memory, has no significant time lag,capable of real-time operation, and capable of automatically producing asharp image for various scene types even when the scene to be capturedis unknown and unpredictable.

REFERENCE SIGNS LIST

-   600 Image capturing device-   602 Pre-capturing module-   604 Object sharpness evaluating module-   606 Focus varying range determining module-   608 Image capturing module-   610 Deconvoluting module-   612 Internal buffer-   801 Image-   805 Image-   809 Image-   811 First object-   812 Second object-   813 Third object-   1202 Optical system-   1204 Image sensor-   1206 ADC-   1208 Image processor-   1210 Microcomputer-   1212 External memory-   1214 Display device-   1216 Storing and/or transmitting device-   1218 OIS sensor-   1220 Driver controller-   1222 Operating unit-   1240 Internal memory-   1242 Raw image processor-   1243 Color image processor-   1244 3D image processor-   1246 Extended depth of field module-   1300 Depth of field extending device-   1302 Lens system-   1304 Image sensor-   1320 Controller-   1340 Memory-   1342 Focus varying range determining module-   1343 Deconvoluting module-   1345 Estimating module-   1346 Object sharpness evaluating module-   1347 Module-   1400 Image capturing device-   1420 Controller-   1424 Memory-   1442 Focus varying range determining module-   1443 Deconvoluting module-   1445 Estimating module-   1446 Object detecting module-   1447 DIBR module-   1500 Image capturing device-   1514 Display module-   1520 Controller-   1524 Memory-   1542 Focus varying range determining module-   1543 Deconvoluting module-   1545 Estimating module-   1546 User interface module-   1547 DIBR module

The invention claimed is:
 1. An image capturing device which captures animaging target while varying a focus position and outputs at least oneoutput image as a capturing result, the image capturing devicecomprising: a pre-capturing unit configured to capture the imagingtarget at each of different focus positions in a predetermined focusrange and output, as a capturing result, a plurality of pre-capturedimages lower in resolution than the output image; a sharpness levelcomputing unit configured to compute a sharpness level of each of thepre-captured images captured by the pre-capturing unit; a focus varyingrange determining unit configured to determine a focus position varyingrange within the predetermined focus range based on sharpness levelscomputed by the sharpness level computing unit, such that asignal-to-noise ratio of the output image is greater than or equal to apredetermined threshold; an image capturing unit configured to capturethe imaging target by sweeping a lens and varying the focus positionduring exposure according to the focus position varying range determinedby the focus varying range determining unit, and output the at least oneoutput image as a capturing result; and an extended depth of field imagederiving unit configured to derive an extended depth of field imagehaving an extended depth of field, using the at least one output imageoutputted by the image capturing unit and at least one predeterminedpoint spread function: wherein the focus varying range determining unitis configured to: divide each pre-captured image into regions; obtain,for each pre-captured image, an in-focus level that corresponds to asharpness level of each of the regions; compare the in-focus levelsbetween corresponding regions in respective pre-captured images; obtain,for each region, a focus position corresponding to a pre-captured imagehaving a highest in-focus level among the in-focus levels of the regionsobtained for each pre-captured image; and determine the focus positionvarying range such that the obtained focus positions are included. 2.The image capturing device according to claim 1, wherein the focusvarying range determining unit is configured to determine the focusposition varying range by determining, among the obtained focuspositions, a nearest position as a start point and a farthest positionas an end point.
 3. The image capturing device according to claim 1,wherein the focus varying range determining unit is configured todetermine a plurality of focus position varying ranges by determining,in each of focus position groups, a nearest position as a start pointand a farthest position as an end point, the focus position groups eachincluding, among the obtained focus positions, focus positionssatisfying predetermined consecutiveness.
 4. The image capturing deviceaccording to claim 1, wherein the image capturing unit is configured to:adjust an exposure time based on a size of the determined focus positionvarying range so as to prevent saturation of pixels of an image to becaptured; and capture the imaging target within the adjusted exposuretime while varying the focus position according to the focus positionvarying range.
 5. The image capturing device according to claim 1,wherein the focus position varying range is limited by a predeterminedexposure time.
 6. The image capturing device according to claim 1,further comprising: an object detecting unit configured to detect anobject in the imaging target; and an object region identifying unitconfigured to identify, based on the detected object, an object regionthat is a partial region of an image obtained by capturing the imagingtarget such that the detected object is at a center of the image.
 7. Theimage capturing device according to claim 6, further comprising aselection receiving unit configured to receive, from a user, selectionof a region of the imaging target in which an object into be captured,wherein the object region identifying unit is configured to identify theobject region based on the selection of the region received by theselection receiving unit.
 8. An image capturing method for capturing animaging target while varying a focus position and for outputting atleast one output image, the image capturing method comprising:pre-capturing the imaging target at each of different focus positionsand obtaining a plurality of pre-captured images lower in resolutionthan the output image; computing a sharpness level of each of thepre-captured images captured in the pre-capturing; determining a focusposition varying range based on sharpness levels computed in thecomputing; capturing the imaging target by sweeping a lens and varyingthe focus position during exposure according to the focus positionvarying range determined in the determining, and outputting the at leastone output image; and deriving an extended depth of field image havingan extended depth of field, using the at least one output imageoutputted and at least one predetermined point spread function, whereinthe determining of the focus position varying range includes: dividingeach pre-captured image into regions; obtaining, for each pre-capturedimage, an in-focus level that corresponds to a sharpness level of eachof the regions; comparing the in-focus levels between correspondingregions of the respective pre-captured images; obtaining, for eachregion, a focus position corresponding to a pre-captured image having ahighest in-focus level among the in-focus levels of the regions obtainedfor each pre-captured image; and determining the focus position varyingrange such that the obtained focus positions are included.
 9. Anon-transitory computer-readable recording medium for use in a computer,the recording medium having a computer program recorded thereon forcausing the computer to execute the image capturing method according toclaim
 8. 10. An integrated circuit which captures an imaging targetwhile varying a focus position and outputs at least one output image,the integrated circuit comprising: a pre-capturing unit which capturesthe imaging target at each of different focus positions and outputs aplurality of pre-captured images lower in resolution than the outputimage; a sharpness level computing unit which computes a sharpness levelof each of the pre-captured images captured by the pre-capturing unit; afocus varying range determining unit which determines a focus positionvarying range based on sharpness levels computed by the sharpness levelcomputing unit; an image capturing unit which captures the imagingtarget by sweeping a lens and varying the focus position during exposureaccording to the focus position varying range determined by the focusvarying range determining unit, and outputs the at least one outputimage; and an extended depth of field image deriving unit configured toderive an extended depth of field image having an extended depth offield, using the at least one output image outputted by the imagecapturing unit and at least one predetermined point spread function,wherein the focus varying range determining unit is configured to:divide each pre-captured image into regions; obtain, for eachpre-captured image, an in-focus level that corresponds to a sharpnesslevel of each of the regions; compare the in-focus levels betweencorresponding regions of the respective pre-captured images; obtain, foreach region, a focus position corresponding to a pre-captured imagehaving a highest in-focus level among the in-focus levels of the regionsobtained for each pre-captured image; and determine the focus positionvarying range such that the obtained focus positions are included.